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Abstract: A light stripe vision system is used to measure the location of poly- 
hedral features of parts from a single frame of video camera output. Issues such as 
accuracy in locating the line segments of intersection in the image and combining 
redundant information from multiple measurements and multiple sources are ad- 
dressed. In 2.5 seconds, a prototype sensor was capable of locating a two inch cube 
to an accuracy (one standard deviation) of .002 inches (.055 mm) in translation 
and .1 degrees (.0015 radians) in rotation. When integrated with a manipulator, 
the system was capable of performing high precision assembly tasks. 
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1 Introduction 

Fast and accurate location of objects is important in many automated industrial 
tasks involving inspection, assembly and machining. Such a measurement capabil- 
ity can change the way certain tasks are currently performed, for instance, force 
controlled motion and precise part fixturing are unnecessary if an assembly system 
can align mating surfaces of parts. 

An interest in automating mechanical assembly tasks prompted a study of sens- 
ing systems which provide computers with information about the position of parts. 
The most critical sensor design requirements for a position-sensor-based assembly 
system are high speed and accurate operation; thus, many of the object ranging 
systems in the literature [3,8,20,21,30,34] cannot be used. 

The system described in this paper operates in a relatively highly constrained 
industrial environment. We assume that the a priori position of a part is known 
approximately; that is, the expected location of the part is available to the sensor 
before it makes a measurement. With this assumption, fairly sparse data may be 
used to obtain a relatively precise measurement. A conceptually similar approach 
was taken by Benton and Waters [5] and Rutkowski et al. [29] who fit a model to 
light stripe data. Their approach depended upon an iterative technique and was 
somewhat slower and less accurate than the technique discussed here, however, it 
could locate a larger variety of part features. 

2 Sensing Method 

A system capable of making measurements in six degrees of freedom is required 
for the part position sensing function. In addition, a flexible (capable of measuring 
many different parts), high speed, high accuracy, and low cost system is desirable. 
Jarvis [24] presents an overview of a number of three-dimensional ranging systems. 
The light stripe ranging technique was selected because it best fits the sensing 
system requirements. 

The light plane of a light stripe system is projected across one or more features 
(such as a corner of a polyhedron or an end of a truncated circular cylinder) of a 
part, Figure 1. Data from an image taken by a video camera positioned at some 
disparate angle with respect to the light plane is processed to locate the feature. 

Triangulation is used to obtain three-dimensional data from the two-dimensional 
sensing element data, Figure 2. A common light source for light stripe systems is 
a laser beam which has been passed through a cylindrical lens. The lens diverges 
the beam in a direction perpendicular to the lens' cylindrical axis. 

The light plane is the sole functional illumination source in the scene as viewed 
by the camera. An isomorphism exists between all points in the image and points 
lying in the light plane in three-space. In order to determine the transformation 




Light Plane 



Camera 



Figure 1: Three line segments generated by the intersection of a plane of light and 
the surfaces of a polyhedral feature may be sensed by a video camera and used to 
locate a part. 

from the two-dimensional image coordinates to accurate three-dimensional space 
coordinates, the system must be precisely calibrated. During the calibration, the 
values of three parameters must be determined which locate the light plane with 
respect to a coordinate frame defined by the camera image plane. A disparity angle, 
LP , a tilt angle, <f> LP , and an offset, vlp, are the parameters used for defining the 
light plane location (these are not unique). 

Discussion in this paper is limited to objects whose surfaces are planar (actually 
only those surfaces being sensed need be planar); thus, the intersection curves 
between the light plane and part surfaces are line segments. 



2.1 Inverse Projection of Image Points 

The coordinates of points in space lying in the light plane may be derived as a 
function of the image plane coordinates and the light plane and camera calibration 
parameters. Some authors have used a matrix description for the geometry of a light 
stripe system [2,26]. Separate equations are maintained here. The transformation 
of a point on an object, (x,y,z), to image plane coordinates, (t/j,£/), is described 
by the perspective projection [35] 
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We may obtain the x coordinate in terms of the y and z coordinates using the light 
plane constraint. From Figure 2 

x = x Q + tan 6 L p (y + z tan <f> LP ) (2) 

where x is the position along the optical axis where the light plane crosses, 

xo = ~Vlp tan 6 LP . 

In general xo < 0. Combining (1), and (2), we obtain 

(3) 
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. (5) 

1 + tan LP (tan <f> L p% + %) 

These equations describe the position of a point in space (x, y, z) as a function of 
the coordinates of the corresponding point in the image plane (t/j,2j). 

2.2 Calculation of Three-Dimensional Vectors from the Light Stripe 
Image 

The measurements obtained from the image of the intersection of a light plane 
and a polyhedral feature is a set of line parameters, [mj^, &/,,], which are defined 
by the equation of the line (refer to Figure 3) 

Vi = muzi + bi t i. (6) 

Using (3), (4), (5), and (6) we may obtain expressions for the vectors along the 
light stripes in real space, /,, and vectors from the origin to the intersection of the 
light stripes and the z = plane, 6j, 
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Figure 4: Light plane intersecting a polyhedral feature and a corresponding model 
of the feature. 

3 Polyhedral Feature Localization 

Most of the light stripe systems in the literature require scanning a complete 
scene and processing a number of video frames. With the light plane positioned 
at a certain orientation with respect to a polyhedron, three or more line segments 
comprise the curve of intersection and only a single frame is necessary to locate the 
part given its approximate a priori position. The following geometric conditions 
are sufficient for three light plane intersection lines to locate a part [32] 

• Pairs of lines must not be parallel. 

• Pairs of faces in which lines lie must not be parallel. 

• The common normal of any pair of lines must not be parallel to the line of intersection 
of any two faces in which the lines lie. 



3.1 Locating General Polyhedral Features 

The position of a polyhedral feature of an object is referenced relative to a 
model of the feature in a base coordinate system, Figure 4. The orientation of the 
feature may be determined by the slopes of three line segments in the image. The 
displacement may be determined from the results of the orientation calculation and 



the intercepts of the three lines; thus, the orientation calculation may be decoupled 
from the displacement calculation. 

3.1.1 Determining the Orientation 

We wish to find the rotation between the model and the object. Assume that 
the three normal vectors for faces A, B, and C of the model in Figure 4 are h A , h B , 
and h c respectively. The normal vectors of the object are unknown, but vectors 
along the intersection line segments on the object are known from the sensed data. 
Assuming that the correspondence between faces is known (approximate a priori 
position), the unit vectors along the intersection line segments for faces A, B, and 
C are l A , l B , and l c respectively. The rotation matrix R from the model to the 
object is found using quaternion notation [23,28,31]. We denote a quaternion by 
the bold face letter, q. Every quaternion has a corresponding four vector, q, whose 
components are the four elements of the quaternion. A quaternion with a zero 
scalar element represents a three vector; thus a quaternion multiplied by a vector 
is defined. A rigid rotation of a vector v is 

qvq* 

where q is a unit quaternion and q* is the quaternion complement of q. 

If the proper rotation is found, the line vectors, /,, will be perpendicular to the 
rotated model normal vectors, q»,q*; that is, the dot product will be zero 

£-qn,q*=0, i = A,B,C. (10) 

Each of these equations are quadratic in components of q. Rewriting this in matrix 
form 

fXiq = 0, i = A,B,C, 

l?1 = 1 (11) 

which is a coupled set of four quadratic equations. Each of the three A, matrices 
(4 x 4) have elements of quadratic polynomials in components of /,- and n k (j and k 
take on values of A, B or C). The components of q may be determined using any of 
a number of numerical methods. The rotation matrix, R, may be calculated from 
the components of q [31] 

I" ?o + 9i - Q 2 - Qz 2(-<fotf3 + 9i q 2 ) 2(q q 2 + qiq 3 ) 
R= 2{q q 3 + qiq 2 ) ql~q\ + ql-ql 2{-q Q q x + q 2 q z ) 
. 2(-9o<72 + <?i<fe) 2(qoqi + q 2 q 3 ) ql - q\ - q\ + q\ 

An alternate approach to locating the general polyhedral feature is to use an 
iterative method such as the one in [5,29]. The assumed o priori location of the 
feature provides a good initial guess and improves solution convergence. 



3.1.2 Determining the Displacement 

Once the rotation from the model to the object is known, the normal vectors of 
the object, h bj,i, may be calculated 

n o6y> ,=Rn t -, i = A,B,C. (12) 

Unless an intersection line is parallel to the 2 = plane, we may define the inter- 
cept vector, bi, which lies in the z = plane from the origin to the i th intersection 
line. The equation of the plane which lies on the t"* face is [15] 

{£-bi)'h objti = (13) 

where x = [x y z] T is the location of a point in space. Solving these three equations 
simultaneously for the planes on faces A,B, and C, we obtain the location of the 
corner 
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where x denotes the vector product and p, is the perpendicular distance from the 
origin to the »'** plane 

Pa = (*a • hobj,A) 
Pb — {f>B " hobj,B) 
Pc — {be -nobj.c)- 

(15) 

3.2 Locating Right Corner Features 

The solution for the rotation of the right polyhedral corner feature may be solved 
in closed form (in contrast to using a numerical technique for general polyhedra). 
We define a right corner feature as the intersection of three half spaces (see Figure 5) 
whose bounding planes are mutually orthogonal. A right-hand coordinate system, 
Jf , with axes xp,yF, %f perpendicular to the bounding planes of the half spaces and 
origin located at the point of intersection of the bounding planes defines the position 
of the feature in six degrees of freedom. We wish to determine the orientation and 
position of this feature reference frame with respect to the world frame, Jw , located 
at the camera image plane. 

3.2.1 Determining the Orientation 

The orientation of the feature reference frame, ? F , with respect to the world 
reference frame, Jw, is determined by locating an intermediate reference frame 
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Figure 5: A right corner feature and reference frames used to determine its orien- 
tation. 



associated with the laser plane, T LP . For the present analysis, we assume that 
the light plane has no width (two-dimensional). We also assume that three line 
segments (no width) are generated when the light plane intersects the feature. The 
orientation of the three line segments are defined by the three vectors I a, Ib, and 
l c as shown in Figure 5. 

We assume the ranging sensor has been accurately calibrated and the unit vector 
normal to the light plane, hip, is known. The intermediate coordinate system, 7lp, 
has axes xlp in direction of one of the lines of intersection (I a), zlp in direction 
hip, and y^p in direction h^p x l A . 

The rotational transformation (in matrix form) from 7w to Tlp is 
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Jlp may be located with respect to Jp by realizing that the cross product of any 

AAA 

two of the three vectors I a, Ib, or lc is normal to the light plane (in the direction 
of zip) and I a lies on the Vf,zf plane. The transformation from fw to J^p is 1 
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where 6 a is the angle between I a and zj?, and 6b is the angle between Ib and yj?. 
The angles 6 a and 0b may be calculated from the angle between I a and lc {6ac) 
and the angle between I a and Ib {Oab) (these angles are both easily obtainable from 
the processed image) 



6a = arctan 
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Multiple values of the rotation matrix, Rjt.lp , which correspond to different light 
stripe image interpretations result from these equations. The proper rotation ma- 
trix is the one closest to the expected nominal rotation matrix. The orientation of 
the feature reference frame, 7p, with respect to the world coordinate frame, Jw, is: 



Tl.W,F — Rw.LpR^lp 



(18) 



x The left superscripts F l denotes that the vector is expressed with respect to the Tp reference 
frame. 
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Description 
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Line target 


i-imiifji: 


Image of £ in the camera's image plane 


AT 


Number of illuminated pixels. 


(A, ft) 


Coordinates of the center of the i"' illuminated pixel 


(s.'.Jfc) 


Coordinates (probabilistic) of points lying on £„„.,,,, 
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Probabilistic distance in the y direction from the 




center of the t"' illuminated pixel to £,„,„,,, 





Orientation angle of £ measured counterclockwise 




from the x axis 


~ y " 


y axis intercept of £ 


». $< 


Least squares estimators for 6 and y t > 




Estimators for 6 and y n from discrete data (£,, y~j) 


A6, Ajft, 


Difference between estimated line parameters (0, y u ) 




and actual parameters [0, t/u) 


L,nz,:t 


Length of each square pixel 


°l 


Variance of 0"t/, for all t = 1, . . . N 


X,-m 


Distance from the center of £,,„„,,,. to the y axis 




measured parallel to the i axis 



Table 1: Nomenclature for line parameter error analysis. 

3.2.2 Determining the Displacement 

Once the rotation is known, the displacement of the corner from the base coor- 
dinate system is given by 14. 

4 Locating Straight Line Features in Quantized Images 

Determining the accuracy of measurements of polyhedral features is studied by 
addressing the more fundamental problem of determining the accuracy of fitting 
two-dimensional line segment parameters. The precision of a least squares technique 
in determining the orientation and position of thin line segments in the presence of 
quantization noise is investigated. The probabilistic technique used in this study is 
similar to the one reported by Hill [22]. 



4.1 Errors in Fitting Linear Parameters to Discretized Data 

Data generated by straight-line features appear as discrete points of various 
intensities located within some width of a central axis in the image plane of a 
discrete array sensor. These points can be processed to find the best fit line through 
them using a least squares technique [1,4]. This section explores the accuracy with 
which straight line features may be found. The variables used in the subsequent 
analysis are listed in Table 1. 




Figure 6: Parameters for the image of a line (infinitely thin). 

4.1.1 Theoretical Errors from Linear Least Squares Approximations 

For the following analysis it is assumed that an image is generated from a single 
line (infinitely thin), £, which lies in a plane parallel to the image plane. The line 
is located at orientation 6 from the horizontal (x) axis and passes over the y axis at 
intercept y Q . For simplification, the projection of £ into the image plane produces 
an image, £ image identical to £ (that is, the angle and intercept remain 6 and y 
respectively - see Figure 6). A typical output from a CCD (charge coupled device) 
video camera (the only type being considered here) is an analog signal constructed 
from a number of intensity readings from discrete pixels in a rectangular grid. For 
this analysis, we assume that the image has been thresholded and transformed into 
a binary array of square elements with length Lpi Xtl . 

For geometric reasons, when e [\ , §tt], one pixel per row is illuminated and 
when 6 e [-f, J], only one pixel for every column is illuminated. We define an 
illuminated pixel as the one in a particular row (or column) whose border circum- 
scribes the longest segment of £ image . Although this definition loses some positional 
information from the array, it simplifies the subsequent analysis. Later in this sec- 
tion corrections from grey level information are suggested which preserve more of 
the available positional information. 

The sensor data used for curve fitting is a list of pixel center coordinates for 
the illuminated elements (x,,&) i = 1, . . . JV. A "best fit" orientation and intercept 
{6 and jfo respectively) may be found in the least squares sense by minimizing the 
sum of the squares of the distances from the best fit line to data points (x t , &•) with 
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Figure 7: Probabilistic location of points on a line. a. Intersection of a column of 
pixels with a line in the range 6 6 [-* , \ ]. b. Probability density function for the 
location of the y coordinate of dmage measured from the center of the illuminated 
pixel. 

distance measured one of three ways: 

A: Minimize distances parallel to the y axis. 

B: Minimize distances parallel to the x axis. 

C: Minimize distances perpendicular to the line. 

Each of these gives a slightly different result for and yo and the best choice is not 
immediately apparent. 

Selection of the most appropriate metric to minimize depends upon the gross 
orientation of L. As a result of our definition of an illuminated pixel, (one pixel 
for every y coordinate is illuminated when G [f,§7r], and one pixel for every x 
coordinate is illuminated when 6 G [-f , f ]), we can choose an x coordinate in the 
first case which is deterministic (x t - will always correspond to a point on the line 
within that pixel - see Figure 7a), and a y coordinate in the second case which 
is deterministic. With these choices of coordinates, we should minimize errors in 
probabilistic coordinates y,- in the first case (metric A) and x, in the second case 
(metric B). 

In the subsequent analysis is assumed to lie in the interval [— J, |]. The least 
squares estimates for orientation 6 and intercept yo are found by minimizing errors 
parallel to the y axis [1,4] 
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where (x,-,y,) are the coordinates of points lying on £ tmo ,«. If (x,-,y,) were known 
precisely, the least squares estimates would be identical to the actual line parame- 
ters; however, in the interval G [— \ , \ ], only precise x,- values are available. 

We are considering the case where x, is deterministic and y,- is probabilistic and 
we desire a metric for the confidence of the estimated line parameters and $/o 
calculated from the measurements (x,, &•). We shall derive the variance of and ffo 
(<7~ and a~ respectively) as a function of all (x,-, yi)'s and the variance of y,- {o\ t )- 

Each random variable y, may be written as 

yi = Ui + hi ( 21 ) 

where y< is deterministic and 6y f - is a random variable with zero mean. At first it 
might appear that the 5y,'s are highly correlated since they all lie on the same line; 
however, some authors [22,27] have suggested that independence between 6y,'s is 
a good assumption for certain cases. In the case of the straight line, the degree 
of correlation depends upon the orientation of the line. At some orientations, the 
values of £y, may change in an unrelated fashion (slightly correlated) while at other 
orientations, the values of 6y,- may exhibit a periodic pattern (highly correlated). 
We make the initial assumption that the Sy^s are independent and a supposition 
that this might not be valid for lines at certain orientations. 

If the maximum absolute value of the £y,'s are small, (19) and (20) can be lin- 
earized about the points (x t , yj) by taking the Taylor series expansion, and keeping 
the first two terms 

S"t+lb(¥) *» ( 22 ) 

where ( £— J is the partial derivative with respect to y t - evaluated at (x,-, y,) and 
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For all 8yi independent of one another [10] 
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Substituting (19) and (20) into (26) and (27) respectively then making the assump- 
tion that all 6yi have the same variance <7 2 , and setting x,- to x,-, we obtain 
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Equations (28) and (29) give the variances of the fitted line parameters as a function 
of the data points (x,-, y.) and the variance of the location of the y coordinate of 
the line in any column containing an illuminated pixel (similar results are obtained 
in [4]). 

If columns y, have an illuminated pixel for all t = 1, . . . JV then the expressions 
for the variance of and yo become 
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where x cm — /L, ' = 1 ' is the calculated center of the line segment in the x direction. 
These results describe the accuracy with which a line may be found from a least 
squares technique as a function of the approximate slope (m), the orthographic 
projection of the length of the line (in pixels) onto the x axis (N), the center of the 
line in the x direction (x cm ), and the variance of the location of the y coordinate 
for any one column of the image (a 2 ). a| does not depend on the position of the 
line segment in the pixel grid, while ai. depends on the distance of the center of 
the line segment from the y axis. For a line (zero width), the probability density 
function for mutually independent 6y,- is a uniform distribution one pixel in width 

L 3 

with a a} y . = ajj = - J ^ L (see Figure 7b). The standard deviations a^ and <r- are 
plotted in Figure 8 as a function of the number of points N for a line positioned 
near = and with x em = y. 
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Figure 8: Standard deviations for orientation and y intercept estimates as a function 
of the number of illuminated pixels. 



The above analysis is valid for a grey scale images of finite width lines as well 
as binary images of thin lines. If yi is permitted to take on subpixel values, and 
grey scale levels are used in an intensity weighted calculation for the center of the 
line, y~i becomes a better estimate of the actual value y,- (a* would be smaller - see 
[17]). The reduction in o* is a function of the width of the line and the resolution 
(in intensity measurements) of the camera. 

In support of the theoretical analysis, computer simulations and experimental 
tests were performed. In both studies, sample statistics from 30 trials were calcu- 
lated and used as a basis for comparisons. Figure 9 shows the results of the study 
with statistics generated in each of eighteen equally spaced orientations from zero 
to forty-five degrees. The sample standard deviations for each set of thirty trials 
(each trial is at a different random orientation within a 2.5 degree range) with a 
line 128 pixels long are plotted for each interval along with the theoretical result, 
(30). A plot of the accuracy of fitted parameters of lines from 45 to 90 degrees 
from metric B would appear as the mirror-image of Figure 9 about the 45 degree 
position. Singularities in A$ occur at slopes near 0, j, |, |, |, and 1 (shown in 
figure). In these areas, measurements of the location and orientation of dmage are 
not as accurate because there is more space for t to translate and rotate before 
pixels are caused to change state, (see Figure 10). The theoretical result does 
not predict these singularities because the mutual independence assumption is not 
valid there. The simulation and experimental sample standard deviations at all 
orientations are, however, theoretically bounded by an analysis assuming perfect 
correlation [18]. 
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Figure 9: Accuracy of the least squares fitting routine for discretized lines as a 
function of their orientation. Theoretical, computer simulation and experimental 
results are shown. 
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Figure 10: Range of motions of lines before change in state of sensor occurs. 



5 Using Redundant Sensed Information 

Techniques for combining similar information from a number of sources or mul- 
tiple measurements from the same source may be used to improve the accuracy of 
a single measurement [6,7,11,33]. The equivalent of four or more line segments may 
be generated on a polyhedral feature either by using two or more images with the 
part or light plane displaced slightly, or by using multiple light planes and a single 
image. In addition to allowing a more accurate calculation of the location of a part 
feature, a system of multiple light planes has the advantage of having illumination 
sources which are not in a single plane; thus, it may be possible to generate longer 
line segments or have line segments in more desirable locations with respect to the 
part feature. In fact, some features may be localized with a number of light planes 
but not with a single light plane. 

5.1 Orientation Estimation for Polyhedral Features Without Measure- 
ment Covariance Information 

This technique does not take the quality of different measurements (covariance) 
into account. A similar problem is addressed by Faugeras and Heebert [12,13,14] 
who find the best fit orientation of an object from a number of sensed normal 
vectors. We were not able to use their technique since our measurement data is 
in the form of line segments on the faces of the feature. With each additional line 
segment, there is an additional equation associated with (10) and we may minimize 
the sum of the squares of the dot products 

min Etf-qn,-q*) 2 . (32) 

»' 

A value for q which minimizes this quartic expression may be found using a nu- 
merical technique. 

5.2 Optimal Linear Estimation of Orientation for a Polyhedral Feature 

In this technique, the covariance of the measurements are taken into account 
and a linear estimator is used to estimate the state (position) of the object. 

5.2.1 Representation of Rotations 

For the subsequent analysis we use an Euler angle description of rotations be- 
cause the number of variables needed to describe a rotation is equal to the number 
of degrees of freedom. The Euler angle convention is shown in Figure 11. A coor- 





Figure 11: Euler angles used in the least squares estimate for the orientation of a 
polyhedral feature. 



dinate frame located at Euler angles 



a= < 



6 " 

{ V> J 



is first rotated about the base system z axis by angle 6 then about its own x axis 
by angle <f>, then about its own z axis by angle $. 



h.1.1 Obtaining an Optimal Linear Estimate from Image Line Slopes 

We assume that orientation estimates expressed in Euler angle notation, {a,}, 
are available. In general, these may be obtained from three appropriately chosen 
line segments. 

*i = fi{Si,uS it2 ,S it 3), t = l,...,M (33) 

— * 

where /, is a vector function, S,,i, S,,2,andS,,3 are the slopes of the three line seg- 
ments from the sensor, and M is the number of different estimates. We wish to 
combine these estimates to arrive at the most accurate estimate possible, a, given 
the expected errors of the measurements. 

The functions /,• are functions of the calibration variables and part model pa- 
rameters as well as the slopes of the line segments. For simplicity, only the errors 
in slopes are considered in the following Kalman filter design. If the magnitude of 
the unmodeled uncertainties are large enough, imprecise results may be generated. 

In general the functions /,• in (33) are non-linear and we cannot directly apply 
linear optimal estimation theory [16]. We can, however, assume that errors in 



slope measurements, £S,-,j, are small and linearize (33). Taking the Taylor series 
expansion and keeping terms of first order or less 



«• « £ + 53 ^~ 6S <,i 



(34) 



where / takes on the appropriate values for the lines used in the estimates and £S,-,j 
is a random variable which is the difference between the actual slope for line / and 
the measured slope. Neglecting higher order terms, (34) in matrix form is 



where 



Ks, = 



a, = £ + K Sii 6Si 
dfi dfi dft 



dSi,i &Si t 2 dSij 



and 6 Si = 



6St,i ) 
6S ii2 



> . 



(35) 



(36) 



The partial derivatives may be calculated directly if an analytical expression is 
available for /,-. If /,• involves a numerical technique, the partial derivatives may be 
approximated by 

d/ t fi{Si,u- • • . {Sjj + e), . . . , S,-, 3 ) 

dS itj ~ e 

where e is a small number whose addition to S,,y will produce a result in the nu- 
merical calculation significantly larger than the numerical error. 
Rearranging terms, (35) becomes 



--i.z 



K Si a i =K s )a i -6$ i , t'=l,...,M. 



(37) 



All M equations in (37) may be combined into a single equation by adding the rows 
of each vector term corresponding to the same line segment 



where 



zs=U s a + vs (38) 

t « 

VS = -J2 S $i' 
i 

The sums are taken over identical lines only so that (38) has the same number of 
rows as independent line slopes in the image. 

Equation (38) is in the linear optimal estimation equation form [16]; thus, the 
maximum likelihood estimate for the Euler angles is 



-IttT/i-1-. 



a = (HJC^H^-'HJCJ 1 ^ 



(39) 



where C s is the covariance matrix of the noise vector v$- This matrix is diagonal 
with elements equal to the variances of the slopes of the respective lines. These 
variances are calculated from (30). 

The minimum variance Bayesian estimate of the Euler angles given an a priori 
estimate a— and initial covariance matrix P~ is [16] 

a = oT ([P-]" 1 + HjC^HfJ-'HjC^ft - H 5 aT) (40) 

which is a discrete Kalman filter. This recursive estimator may be applied to 
position information from fundamentally different sources so long as a measure of 
the expected accuracy (covariance matrix) of the information is available. 

6 Results of Prototype Sensor Accuracy Tests 

A prototype light-stripe-part-position sensor mounted in a test bed arrangement 
is shown in Figure 12. The sensor consists of an MOS-type camera with 320 X 244 
pixels and a laser-cylindrical lens light source. The test bed has three translational 
stages and three rotational stages. 

The measurement accuracy of the test-bed-mounted sensor was studied by mea- 
suring the corner feature of a cube, Figure 13. Corner location measurements were 
compared to the settings on the six stages of the test apparatus. These measure- 
ments were relative to an arbitrary reference location. It was not necessary to 
obtain absolute measurements since relative accuracy is the specification required 
to determine sensor performance in an assembly environment where all measure- 
ments are relative to the robot frame. 

The relative accuracy in locating the corner of the cube using information from 
two light planes was also studied. The maximum likelihood technique developed in 
Section 5 was used to obtain estimates of the orientation of the cube. 

6.1 Test Procedure 

The test procedure for determining the accuracy of the sensor entailed locating 
the cube at a home position then moving one of the six positioning stages (call this 
the test direction) a small amount. An image of the cube was recorded and the 
same stage was moved the same amount once again. This was repeated a number 
of times. The positions of the cube corner from a series of images were compared 
to the apparatus positions and sample statistics were calculated from differences 
between stage motion and sensed corner position. The test procedure was repeated 
for all six stages. The rotational degrees of freedom in stage motion, x rot , <f> rot , and 
Vv«,t are defined in Figure 13. Measurements were made in the 6, <j>, and xj> degrees 
of freedom, which are the Euler angles of the cube. Figure 11 shows the convention 
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Figure 12: Test bed used for sensor accuracy studies. 




Figure 13: Test cube mounted on rotational stages. 



Direction of 
Motion 


Laser 
Number 


X error 
(inches) 


Y error 
(inches) 


Z error 
(inches) 


6 error 
(degs.) 


<f> error 
(degs.) 


V> error 
(degs.) 


Trans. X 
Trans. X 


1 
2 


.0030 


.0010 
.0025 


.0010 
.0015 


.05 
.13 


.03 
.04 


.08 
.11 


Trans. Y 
Trans. Y 


1 
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.0035 
.0080 


.0025 
.0025 


.0015 
.0025 


.12 
.12 


.09 
.05 


.14 
.13 


Trans. Z 
Trans. Z 


1 
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.0010 
.0025 


.0020 
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.0025 
.0015 


.11 
.15 


.07 
.06 


.13 
.14 


Rot. <f> ro t 

Rot. (j> TO t 


1 
2 






.0015 
.0015 




.05 
.09 


.08 
.08 


Rot. ll> rot 
Rot. Vrot 


1 
2 




.0020 
.0020 




.09 
.14 




.12 
.08 


Rot. x ro t 
Rot. x rot 


1 

2 


.0015 
.005 













Table 2: Results of the part position sensor accuracy tests. Each number is the 
sample standard deviation from 21 images. Mean sample standard deviations are: 
translation - .002 inches (.055 mm), rotation - .095 degrees (.0015 radians). 

used. Since the cube was fairly accurately aligned to the axes of the positioner, <f> rot 
corresponds closely to <j> and ijj r0 t corresponds closely to ip. 

The range of motions for the tests were 0.5 inches (12.7 mm) in each translational 
degree of freedom and 8.5 degrees (.15 radians) for the <f> rot rotation and tp rot rotation 
and 20 degrees (.35 radians) for the x rot rotation. In general, measurements near 
the center of the ranges of motion were more accurate than those near the ends. 
This was because line segments on some cube faces become smaller as the cube was 
moved from a central position under the laser stripe. 



6.2 Single Light Plane Test Results 

Table 2 shows the results of the tests for motions of each of the six degrees of 
freedom. The blank spaces in the table correspond to degrees of freedom which 
were non-stationary or were not linearly related to the test direction motion. The 
mean of the sample standard deviations are .002 inches (.05 mm) in translation and 
0.095 degrees (0.0015 radians) in rotation. The translational error of .002 inches 
(.05 mm) is equivalent to about ^ of a camera pixel. Assuming the distribution of 
errors are Gaussian, 98.8 percent of the measurements (5 standard deviations) will 
be within an accuracy of .010 inches (.25 mm) and .5 degrees (.007 radians). Each 
measurement of the position of the cube took about 2.5 seconds. Much of this time 
was during an image convolution step used to more accurately locate the centers 
of the line segments [17]. 

The repeatability of the sensor was extremely high (about .0001 inches or .0025 
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Figure 14: The intersection of two independent light planes with a cube. 

mm per standard deviation) when multiple readings were taken without moving 
the target in between readings. If the target position was perturbed slightly, the 
readings changed more appreciably (about .001 inches or .025 mm per standard 
deviation). 

In addition to the accuracy tests performed on the testbed, experiments involv- 
ing assembly of mechanical components using the sensor and a PUMA 600 robot 
were performed. The sensor-based assembly system was capable of reliably assem- 
bling a 36 pin electrical connector, a die cast box, and a 1.75 inch (44.5 mm) square 
peg-in-hole with a .004 inch (.1 mm) clearance [17,19]. 

6.3 Multiple Light Plane Test Results 

The accuracy in locating the cube's corner using two light stripes was investi- 
gated. Six line segments (three from each laser) were generated for each position of 
the cube, Figure 14. Two independent orientation estimates (Euler angle vectors) 
were obtained from the line segments and combined using a maximum likelihood 
technique, (38). Two sets of results were obtained: one with all six estimates being 
used in a single maximum likelihood calculation, and one with only pairs of corre- 
sponding Euler angles being used. When pairs of single Euler angles are used in 
the estimate, statistical correlations with other Euler angles are ignored. Figure 15 
shows the Euler angle estimates for lines from laser number 1, laser number 2, and 
the maximum likelihood estimate using all six angles. The calculated angles from 
each of the lasers have different means due to calibration errors. The maximum 
likelihood estimate does not always fall between the two calculated angles and for 
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Figure 15: Maximum likelihood estimates of rotation angles of the test cube. 
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Figure 16: Maximum likelihood estimates of rotation angles of test cube from pairs 
of angles. Plots are for Euler angles 6, <j> and ip. 

some angles seems to be an imprecise estimate. This is most likely due to the cou- 
pling between the Euler angles and the unmodeled calibration uncertainties. The 
maximum likelihood estimates from pairs of angles are shown in Figure 16. Here 
the maximum likelihood estimate is between the two independent laser estimates. 
The estimate nears one of the single laser estimates when that one is more reliable 
(longer line segments) than the other. The estimate always uses the combination of 
the measurements which reduces the overall error. Since calibration errors were not 
taken into account, the estimate weighs the two calculations equally for equal length 
line segments. Because the maximum likelihood estimate performs a weighted av- 
erage of the two estimates, and the two estimates have different means, the sample 
standard deviation of the maximum likelihood estimate is not always less than the 
smaller sample standard deviation of the single laser estimates; although, the tests 
showed that it was usually very close to the lower standard deviation of the two. 



7 Summary and Conclusions 

A light stripe based vision sensor for locating polyhedral features of parts from a 
single video frame has been developed. The system locates features to an accuracy 



of .002 inches (.05 mm or about j of a camera pixel) in translation and about 0.1 
degrees (0.0015 radians) in rotation. The measurements take about 2.5 seconds. 
This may be speeded up to about 1 second if a different hardware image convolver 
is used. 

The calculation for locating general polyhedra requires a numerical computa- 
tion. If the three planes of the feature are orthogonal, a closed form solution for 
the feature location may be used. The accuracy of the feature locating technique 
depends on the lengths of the line segments generated from the intersection of the 
light plane and the part's planar surfaces. The accuracy of fitting lines to data from 
quantized images was found as a function of the length, orientation, and position 
of the segments. 

Two techniques for using redundant data from multiple light planes were dis- 
cussed. One technique may be used to combine measurement data from sources 
in addition to the light stripe sensor. The covariance of a measurement from the 
light stripe system is a function of the variance in locating the line segments in the 
image; thus, the optimal linear estimate weighs those estimates with the longest 
length segments more than those with shorter length segments. 

The sensing system is useful for a number of industrial tasks and was demon- 
strated to be practical for detecting misalignments between parts prior to their 
assembly by a manipulator. 

Non-polyhedral features generate non-linear curves of intersection and a method 
of determining the covariance matrix for the parameters of more general curves is 
required if they are to be incorporated into the optimal estimation technique pre- 
sented in Section 5. More work is required in sensing non-polyhedral features 
and analyzing non-linear curves in light stripe images. Bolles and Fischler [9] and 
Lozano-Perez, Grimson, and White [25] report difficulty in fitting conic sections 
to light stripe data. Ellipse fitting, from intersection curves of cylindrical features, 
gives significantly less accurate results than the line fitting techniques for polyhe- 
dral features. We are investigating multiple light plane intersections of cylindrical 
features which produce additional constraints which could be used in the fitting 
procedure to give better results (see [25] for a discussion of multiple light plane 
techniques for fitting cylinders). 
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